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One of the dilemmas faced by todays faculty is assuring quality in online courses. As one solution 
to that dilemma, Quality Matters (QM), a program of MarylandOnline, built a rubric of design 
standards informed by existing research literature and best practices. The rubric was implemented 
within a faculty-centred, peer review process in which colleagues share their expertise to facilitate 
course design improvements and to achieve an established level of quality in online course design. 
This article will describe the basic tenets and processes ofQM as an inter-institutional quality assur¬ 
ance program for online learning. 


Introduction 

Q uality in online courses need not be caught 
in the riptides that can result from the con¬ 
tinuing waves of educational refocusing. While 
online learning might appear to be a new wave, it 
is not. It has become the most accepted form of 
distance education, bringing along with it legions 
of faculty inexperienced in designing and teaching 
in distance formats. That was the dilemma faced 
in the early 2000s by members of MarylandOn¬ 
line (www.marylandonline.org), a voluntary con¬ 
sortium of community colleges and universities in 
Maryland. 

A definition of distance education seems in 
order. Moore and Kearsley’s (1996, 2005) widely 
cited definition points out that distance education is 


much more than piping traditional classroom courses 
through a communication conduit, such as the In¬ 
ternet. In fact, they define distance education as 
“planned learning that normally occurs in a different 
place from teaching, requiring special course design 
and instruction techniques, communication through 
various technologies, and special organizational and 
administrative arrangements” (2005, p. 2). It is a 
system of education focused on enabling and sup¬ 
porting selected learner-teacher, learner-content, and 
learner-learner interactions via a communication 
technology (Moore, 1989). 

Attempts to assure and record quality in dis¬ 
tance education have a long history. Distance edu¬ 
cation has been documented by Russell (2001) as 
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far back as 1928 as not being significantly different 
than education delivered in traditional classroom- 
based settings. Russell assembled and continues to 
post research findings on the http://nosignificantdif- 
ference.org, website, including no significant differ¬ 
ences in performance on graded assignments, final 
course grades, and other assessments of learning. It 
can be noted that some studies indicated significance 
in favor of either face-to-face or at a distance, but 
most indicated similar results when the two delivery 
formats were compared. 

Recently, the US Department of Education 
(2009) conducted a meta-analysis of existing research 
on the effectiveness of online learning. The final 
report noted that, “on average, students in online 
learning conditions performed better than those re¬ 
ceiving face-to-face instruction.” It was further noted 
that students in “blended conditions” (combining 
online and face-to-face) benefitted the most because 
of “additional learning time and instructional ele¬ 
ments” (p. ix). The meta-analysis methodology used 
was limited to studies using random-assignment or 
controlled quasi-experimental design in web-based 
instruction. Only studies that examined “objec¬ 
tive measures of student learning” were included in 
the final meta-analysis. Studies which focused “on 
student or teacher perceptions of learning or course 
quality” for example, were not included in the analy¬ 
sis (p. xii). The report cautioned “in generalizing to 
the K-12 population” since the 44 studies that met 
the study criteria came from adult learning settings, 
such as higher education or medical training (p. xii). 

Quality Matters 

As online formats were beginning their meteoric rise 
in popularity at the turn of the millennium, mem¬ 
bers of MarylandOnline faced the dilemma of assur¬ 
ing faculty of the quality of shared online courses. 
Course sharing and collaboration amongst consor¬ 
tium member community colleges, four-year colleg¬ 
es, and universities across the state of Maryland had 


been the motivation for organizing MarylandOnline 
in the mid-1990s. 

As sharing of online courses became popu¬ 
lar, faculty of MarylandOnline member institutions 
expressed concern about determining the quality of 
an ‘adopted’ course from another institution. As a 
result, Quality Matters (QM) 1 was envisioned as an 
inter-institutional peer review process to improve the 
quality of online courses. Grant funding from the 
Fund for the Improvement of Postsecondary Educa¬ 
tion (FIPSE), a federal grant of the US Department 
of Education, allowed for development of the project 
from 2003-2006. Since 2006, the program contin¬ 
ues to grow as a not-for-profit, subscription-based 
program and now includes more than 365 subscrib¬ 
ing institutions across the US (and the University of 
Guelph in Canada). 2 

At the 2009 Society for Teaching and Learning 
in Higher Education conference in New Brunswick, 
Alex Lowry kicked off the discussion of dilemmas 
and problem-solving in todays education by present¬ 
ing a 2 x 2 problem/solution dilemma matrix. While 
it is not the purpose of this article to critique QM’s 
development from the Lowry model, I suggest that 
distance educator members of MarylandOnline en- 
visioned a solution to the challenge of assuring quality 
in online courses. By using the work of leaders from 
research and best practices in the field, they developed 
a niche program focusing on course design and peer 
review for the improvement of online learning. 

The Quality Matters Rubric ™ 

Course design - the forethought and planning of 
an online course - includes selection of learning 
objectives, technology and media options, instruc¬ 
tional strategies, and evaluation techniques (Moore 
& Kearsley, 2005; Shearer, 2007). Course design is 
one of many factors affecting online course quality. 
Other systemic factors include: course delivery (the 
way a course is taught); course content; the learning 
management system and its functionality (including 


1 For more information on the program, visit www.qualitymatters.org. 

2 For the complete list of institutions visit http://www.qualitymatters.org/Documents/Subscriber%20List%20for%20Publication.pdf 




Quality Matters 


51 


technical support); the institutional infrastructure 
(for example, help desk, online library access, on¬ 
line tutoring access); instructor training; and learner 
readiness. The developers of QM faced the complex 
dilemma of assuring a level of quality by keeping 
their focus on course design. A faculty-centred set of 
values promoting continuous quality improvement 
guided efforts for faculty to improve their own on¬ 
line courses (with help from instructional designer 
when available). The resulting peer process was to be 
guided by a set of standards that would become the 
QM Rubric™. 

The QM Rubric ™ was developed in its cur¬ 
rent format in 2004 during the FIPSE grant period. 
Design principles put forward by Chickering and 
Ehrmann (1996) and the American Council on Edu¬ 
cation’s Guiding Principles for Distance Learning in a 
Learning Society (1996) and Distance Learning Evalu¬ 
ation Guide (2001) were piloted and consolidated 
(Shattuck, 2007). The aim was that the rubric would 
provide a practical definition and workable tool for 
determining a level of quality in online course design. 
The rubric was built upon best practices, existing dis¬ 
tance education research, and instructional design 
principles (Sener & Shattuck, 2006). 1 2 3 

The Rubric continues to be a living docu¬ 
ment that is reviewed and improved every two or 
three years following the same rigorous approach 
which has become the guiding principles of QM: 
collegial, collaborative, continuous, and centred in 
an academic foundation around student learning. 
For example, as a result of that process, the 2006- 
2008 version of the rubric added online components 
of blended courses. 

The 2008-2010 version of the rubric includes 
eight general and 40 specific standards. 4 The eight 
general standards include: 

1. Course overview and introduction 

2. Learning objectives 

3. Assessment and measurement 


4. Resources and materials 

5. Learner engagement 

6. Course technology 

7. Learner support 

8. Accessibility 

Some examples of the standards include: 

• Standard 1.1: Instructions make clear how to 
get started and where to find various course 
components. 

• Standard 2.1: The course learning objectives 
describe outcomes that are measurable. 

• Standard 3.1: The types of assessments se¬ 
lected measure the stated learning objectives 
and are consistent with course activities and 
resources. 

• Standard 5.1: The learning activities promote 
the achievement of the stated learning objec¬ 
tives. 

Each of the 40 standards has an assigned point value 
(3=essential, 2=very important, 1 important). An in¬ 
teractive rubric tool which includes annotations and 
other explanations to assist faculty reviewers during a 
review is available to QM subscribers. 

Courses under review meet QM standards by 
attaining an overall 85% score, including successfully 
meeting all of the 17 essential standards. The web- 
based, interactive rubric tool calculates and combines 
all scores and comments from the team. A detailed 
final report is presented directly to the faculty mem¬ 
ber whose course was reviewed. 

The Quality Matters Process 

The values of collegiality, collaboration, and academ¬ 
ic foundations related to the impact of instructional 


3 Citations, resources, and referenced best practices can be found at http://www.qualitymatters.org/Documents/ Matrix%20of%20 
Research%20Standards%20FY0506.pdf 

4 A listing of the 40 standards can be found at http://qminstitute.org/home/Public%20Library/About%20QM/ RubricStan- 
dards2008-2010.pdf 
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design on student learning are promoted by a course 
review process in which a faculty member seeks rec¬ 
ommendations for improvement by a team of three 
online teaching peers who have received training 
on applicadon of the QM Rubric™. The Rubric 
is a guideline to help experienced online instructors 
determine if a specific course meets QM standards. 
But, it is more than a review tool. The Rubric is 
part of a process in which peer reviewers make com¬ 
ments and recommendations for improvement of the 
course design. This can be considered from a con¬ 
structivist lens as use of a set of guidelines by peers 
to guide discussion and offer recommendations to a 
colleague on specific course design improvements to 
enhance student learning. 

Review teams are comprised of three mem¬ 
bers, including a subject matter expert (a faculty 
member who teaches in the same discipline as the 
course to be reviewed), a faculty member who is ex¬ 
ternal to the institution which houses the course to 
be reviewed, and a QM master reviewer who chairs 
the review. The faculty member whose course is to 
be reviewed is also considered as an associate of the 
team. She or he provides initial information about 
the course and is encouraged to request focused at¬ 
tention on problematic areas in the course. Members 
of the review team are granted access to the online 
course and encouraged to advocate for students by 
taking a student view of the course. This is usu¬ 
ally an enlightening experience for faculty who find 
themselves assuming a student perspective when en¬ 
tering and navigating an unfamiliar online course. 
Routinely, members of review teams comment on 
the benefits they gain from approaching an online 
course from a student’s perspective while reviewing 
a peer’s course. 

The QM process is twofold: first, to make 
a decision on the current course design based on 
meeting QM quality standards; and second, to make 
comments and recommendations for improvement. 
A key activity of a QM review is the crafting of com¬ 
ments and recommendations by each team member 
as she or he looks for evidence for each of the 40 stan¬ 
dards. Comments and recommendations for further 
improvement are to be made even when a standard is 
met at the QM threshold. The QM review process is 


not an evaluation of the teaching of the course, but 
a method of establishing and improving the quality 
of the design of the course. Evaluation of teaching 
is an institutional responsibility, not one for inter- 
institutional peer review. 

The review team’s recommendations are 
key to the success of a review. Commenting on the 
strengths of the course as well as the weaknesses is 
expected. Training prior to being certified as a QM 
peer reviewer emphasizes that recommendations 
should be constructive, specific, measurable, sensi¬ 
tive, and balanced in a professional, collegial fashion. 
For example, a recommendation is more collegial if 
written in the following way: 

The learning objectives in unit 2 were well 
written - specific, measurable, etc. I did 
however have trouble connecting the as¬ 
sessments to the learning objectives in that 
unit. I think you’ll want to review the 
learning objectives and make sure that they 
are reflected in the assessments. 

This is more constructive and probably better re¬ 
ceived than commenting - “poor assessments” - to 
a professional colleague. All courses submitted for a 
QM review are expected to meet QM standards, ei¬ 
ther initially or upon revision. This confirms that the 
goal is for online faculty to assist peers in improving 
the design of their fully online or online components 
of blended courses. 

Summary 

In response to the challenges presented by the growth 
of online distance education and of emerging con¬ 
cerns of quality from faculty, QM, a faculty-focused 
program of peer review with the goal of improving 
course design of fully online and online components 
of blended courses, was developed as one solution. 
Built on the work of leaders in distance education 
in the form of existing distance education research, 
principles of instructional design, and best practices, 
the QM Rubric™ guides a team of faculty who teach 
online and have completed training in applying the 
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rubric in a collegial manner to others’ courses. De¬ 
veloped by visionaries as a regional solution to the 
dilemma of establishing quality assurance in online 
courses, the program finds a niche in addressing a 
dilemma shared by others. As a living set of tools and 
processes, QM provides some common language and 
standards for online faculty to improve and establish 
a level of quality in their own online courses. 
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